Image processing device and image processing method, imaging apparatus, and computer program

ABSTRACT

Correction processing of crosstalk amount is adaptively performed even under conditions where optical conditions, such as a lens to be used, are unknown. 
     Relative change among crosstalk amount is calculated based on a proportion of the sum of signal amount of RGB signals as to the signal amount of white signals. Next, upon calculating crosstalk correction coefficients corresponding to the current crosstalk amount from a relational expression of crosstalk correction coefficients obtained beforehand and crosstalk amount, the signal of a pixel to be corrected is corrected by subtracting several tenths of adjacent pixel signals from the signal of the pixel to be corrected, as crosstalk amount.

TECHNICAL FIELD

The present invention relates to an image processing device and imageprocessing method, an imaging apparatus, and a computer program, whereinoutput signals from an imaging device having a color filter with colorcoding are processed, an particularly relates to an image processingdevice and image processing method, an imaging apparatus, and a computerprogram, wherein output signals from an imaging device using whitepixels in color coding are processed.

BACKGROUND ART

The camera has a long history as a means to record visual information.As of recent, digital cameras which perform digital encoding of imagescaptured with a solid state imaging device such as a CCD (Charge CoupledDevice) or CMOS (Complementary Mental-Oxide Semiconductor) or the likehas become widespread, replacing silver-salt cameras which take picturesusing film or photosensitive plates. Digital cameras are advantageous inthat images subjected to digital encoding are stored in memory and imageprocessing and image management can be performed by computer, andfurther, that there is no problem of the life expectancy of film.Currently, many digital still cameras, digital video cameras, digitalcameras implemented in cellular phones and PDAs (Personal DigitalAssistants), and monitoring cameras, use solid state devices.

Either imaging devices of CCD and CMOS are configured with anarrangement wherein two-dimensionally arrayed pixels (photodiodes) usephotoelectric effect to convert light into electric charge. The surfaceof each pixel has a color pixel of one of three colors of R (red), green(G), blue (B), for example, and signal charge corresponding to theamount of incident light passing through each color filter isaccumulated in each pixel. The color filters are band-pass filters whichpass light of a predetermined wavelength. Signal charges according tothe amount of incident light of each color are read out from each pixel,and the color of incident light at each pixel position can be reproducedfrom the amount of signal charge of each of the three colors.

As of recent, with the advance in miniaturization technology, higherresolution of imaging devices has advanced. However, miniaturizingpixels due to high resolution leads to the concern that sensitivity willdecrease due to decrease in the amount of charge accumulated at eachpixel. One method that has been proposed to realize high sensitivity iscolor coding of an array including “white (WHITE) pixels) that do notinclude an optical band-pass filter on the pixel (e.g., see NPL 1). Highsensitivity pixels such as white pixels have a feature that thesensitivity to incident light is higher as compared to chromatic pixels,and the sensitivity properties can be improved in a low-illuminanceenvironment (e.g., see PTL 1).

FIG. 12A illustrates a Bayer array which is a representative filterarray of primary colors. Also, FIG. 12B illustrates an example of afilter array including white pixels. Here, in the drawing, R representsRed (red) color filters, G represents Green (green) color filters, Brepresents Blue (blue) color filters, and W represents White (white)color filters, respectively. In the example illustrated in the drawing,white pixels are introduced between the RGB primary color system colorfilters in an intermittent manner.

Also, with miniaturization of pixels, there is concern that optical andelectrical crosstalk, i.e., color mixing (hereinafter referred to simplyas “crosstalk”) will occur between adjacent. Factors of crosstalkinclude leaking of light which should be collected at the adjacentpixel, electrons leaking between pixels, and so forth.

Crosstalk leads to deterioration in resolution and loss of colorinformation, and accordingly needs to be corrected. Now, crosstalk isnot a problem unique to imaging devices using color filters includingwhite pixels in the array. However, a greater amount of light leaks fromwhite pixels, so deterioration of images due to crosstalk is more markedas compared to imaging devices using color filters not including whitepixels in the array.

Even with the same imaging device, the amount of crosstalk variesdepending on optical conditions such as individual micro-lenses. This isbecause crosstalk is dependent on the incident angle. Accordingly, theamount of crosstalk differs depending on the position of the pixels onthe chip face. Also, the depth of penetration into the silicon (Si)substrate configuring the imaging device differs depending on thewavelength of the light, so the amount of crosstalk also changesdepending on the color temperature of the light source at the time ofshooting.

For example, a signal processing method has been proposed which handleschange in crosstalk owing to optical conditions, by performingcorresponding processing as to signals of a pixel of interest usingsignals of each of multiple surrounding pixels adjacent to a pixel ofinterest of the imaging device, and correction parameters setindependently for each of the signals (e.g., see PTL 2). However, withthis signal processing method, the values of the correction parametersare set in accordance with the aperture of the diaphragm included in theoptical system guiding light from the subject to the imaging device.That is to say, the lens to be used is already decided, the amount ofcrosstalk according to the lens has been measured beforehand, andcorrection is performed as to this. Accordingly, correction of theamount of crosstalk is difficult with a situation where lens informationis unknown, such as with exchangeable lenses wherein the user can freelyexchange lenses.

Citation List Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No.2007-288490

PTL 2: Japanese Unexamined Patent Application Publication No.2007-142697

Non Patent Literature

NPL 1: Y. Egawa, “A White-RGB CFA-Patterned CMOS Image Sensor with WideDynamic Range” (2008 IEEE International Solid-State Circuits Conference(ISSCC) P. 52-53)

SUMMARY OF INVENTION Technical Problem

It is an object of the present invention to provide an excellent imageprocessing device and image processing method, imaging apparatus, andcomputer program, wherein output signals from an imaging device usingwhite pixels in color coding can be suitably processed.

It is another object of the present invention to provide an excellentimage processing device and image processing method, imaging apparatus,and computer program, which can suitably perform correction processingof the amount of crosstalk included in output signals from an imagingdevice using white pixels in color coding.

It is a further object of the present invention to provide an excellentimage processing device and image processing method, imaging apparatus,and computer program, which can suitably perform correction processingof the amount of crosstalk included in output signals from an imagingdevice using white pixels in color coding, even in a situation whereinoptical conditions, such as a lens to be used, and so forth, areunknown.

Solution to Problem

The present application has been made in consideration of the aboveproblems, and an invention according to Claim 1 is an image processingdevice including:

a crosstalk amount calculating unit for calculating an evaluation valueof crosstalk amount included in an output signal from a pixel to becorrected in an imaging device;

a crosstalk correction coefficient calculating unit for calculating acrosstalk correction coefficient based on the evaluation value outputfrom the crosstalk amount calculating unit; and

a crosstalk correcting unit for eliminating crosstalk amount included inan output signal of the pixel to be corrected, using the crosstalkcorrection coefficient.

With an invention according to Claim 2 of the present invention, thecrosstalk amount calculating unit of the image processing deviceaccording to Claim 1 is configured to calculate the evaluation value ofcrosstalk amount included in an output signal of the pixel to becorrected, based on output signals from the imaging device.

With an invention according to Claim 3 of the present invention, thecrosstalk amount calculating unit of the image processing deviceaccording to Claim 1 is configured to calculate the evaluation value ofcrosstalk amount included in an output signal of the pixel to becorrected, based on the relation of output signals between adjacentpixels.

With an invention according to Claim 4 of the present invention, withthe image processing device according to Claim 1, the imaging device isconfigured to use color coding including white pixels. Crosstalk is nota problem unique to imaging devices using color filters including whitepixels in the array, but a greater amount of light leaks from whitepixels, so deterioration of images due to crosstalk is more marked ascompared to imaging devices using color filters not including whitepixels in the array. In such a case, the crosstalk amount calculatingunit according to Claim 4 may be configured to calculate an evaluationvalue for crosstalk amount included in an output signal of a pixel to becorrected, based on the proportion of the sum of the signal amount ofthe pixels other than white, as to the signal amount of white pixels, asdescribed in Claim 5.

More specifically, as described in Claim 6 of the present application,the crosstalk amount calculating unit according to Claim 4 may beconfigured to calculate an evaluation value for the relative amount ofcrosstalk included in an output signal of a pixel to be corrected, basedon the proportion of the sum of values obtained by multiplying thesignal amounts of each of RGB pixels by respective predeterminedcoefficients (α, β, γ), as to a value obtained by multiplying the signalamount of white pixels by a predetermined coefficient (ε).

With an invention according to Claim 7 of the present invention, thecrosstalk amount calculating unit according to Claim 4 is configured tocalculate an evaluation value of crosstalk amount, with N×N pixels as anincrement of processing (where N is a positive integer).

With an invention according to Claim 8 of the present invention, theimage processing device according to Claim 7 is configured furtherincluding memory for storing evaluation values which the crosstalkamount calculating unit has calculated in increments of processing,wherein the crosstalk correction coefficient calculating unit and thecrosstalk correction unit respectively perform calculation of correctioncoefficients and correction of crosstalk, using the evaluation valuescalculated using previous frames saved in the memory.

With an invention according to Claim 9 of the present invention, thecrosstalk correction coefficient calculating unit of the imageprocessing device according to Claim 1 is configured to calculatebeforehand a relational expression between the evaluation value ofcrosstalk amount calculated by the crosstalk amount calculating unit,and correction coefficients, and at the time of an evaluation valueoutput from the crosstalk amount calculating unit being output,references the relational expression and calculates a correctioncoefficient corresponding to the evaluation value.

With an invention according to Claim 10 of the present invention, thecrosstalk correction unit of the image processing device according toClaim 1 is configured to subtract, from an output signal of a pixel tobe corrected, a value obtained by multiplying the output signal of apixel adjacent to the pixel to be corrected by the correctioncoefficient, thereby eliminating amount of crosstalk.

With an invention according to Claim 11 of the present invention, theimaging device of the image processing device according to Claim 1 hasdisposed a plurality of arrays for calculating evaluation valuesincluding white pixels, in an array not including white pixels.

With an invention according to Claim 12 of the present invention, thecrosstalk amount calculating unit of the image processing deviceaccording to Claim 11 is configured to use each of the arrays forcalculating evaluation values to calculate evaluation values ofcrosstalk amount occurring at relevant positions. Also, the crosstalkcorrection coefficient calculating unit is configured to calculate acrosstalk correction coefficient based on the evaluation value outputfrom the crosstalk amount calculating unit, for each position where anarray for calculating evaluation values is disposed. Also, the crosstalkcorrecting unit is configured to perform correction of crosstalk using arelevant coefficient, within an array for calculating evaluation values,and performs correction of crosstalk using a crosstalk correctioncoefficient determined based on an evaluation value of crosstalk amountobtained from a nearby array for calculating evaluation values, in aregion outside of an array for calculating evaluation values.

Also, an invention according to Claim 13 of the present application isan image processing method including:

a crosstalk amount calculating step for calculating an evaluation valueof crosstalk amount included in an output signal from a pixel to becorrected in an imaging device;

a crosstalk correction coefficient calculating step for calculating acrosstalk correction coefficient based on the evaluation value output inthe crosstalk amount calculating step; and

a crosstalk correcting step for eliminating crosstalk amount included inan output signal of the pixel to be corrected, using the crosstalkcorrection coefficient.

Also, an invention according to Claim 14 of the present application isan imaging apparatus including:

an imaging device including a color coding color filter; and

a signal processing unit for processing output signals of the imagingdevice;

wherein the signal processing unit includes

-   -   a crosstalk amount calculating unit for calculating an        evaluation value of crosstalk amount included in an output        signal from a pixel to be corrected in the imaging device,    -   a crosstalk correction coefficient calculating unit for        calculating a crosstalk correction coefficient based on the        evaluation value output from the crosstalk amount calculating        unit, and    -   a crosstalk correcting unit for eliminating crosstalk amount        included in an output signal of the pixel to be corrected, using        the crosstalk correction coefficient.

Also, an invention according to Claim 15 of the present application is acomputer program described in a computer-readable format so as toexecute processing of output signals from an imaging device having acolor coding color filter, the computer program causing the computer tofunction as:

a crosstalk amount calculating unit for calculating an evaluation valueof crosstalk amount included in an output signal from a pixel to becorrected in the imaging device,

a crosstalk correction coefficient calculating unit for calculating acrosstalk correction coefficient based on the evaluation value outputfrom the crosstalk amount calculating unit, and

a crosstalk correcting unit for eliminating crosstalk amount included inan output signal of the pixel to be corrected, using the crosstalkcorrection coefficient.

The computer program according to Claim 15 of the present applicationdefines a computer program described in a computer-readable format so asto realize predetermined processing on a computer. In other words, byinstalling the computer program according to Claim 15 of the presentapplication into a computer, cooperative effects are manifested on thecomputer, whereby operation effects the same as with the imageprocessing device according to Claim 1 of the present application can beobtained.

Advantageous Effects of Invention

According to the present invention, an excellent image processing deviceand image processing method, imaging apparatus, and computer program,wherein the crosstalk amount included in output signals of an imagingdevice using white pixels in color coding can be suitably corrected,even under conditions where optical conditions, such as the lens beingused, are unknown, can be provided.

With the invention according to Claims 1, 13, and 15 of the presentapplication, even if there is no optical information while shooting, anevaluation value for telling the crosstalk amount can be calculated fromthe shooting data alone, and correction processing of crosstalk can beperformed using correction coefficients applied to this evaluationvalue.

With the invention according to Claims 2 and 3 of the presentapplication, crosstalk amount is calculated based on output signals fromthe imaging device, so crosstalk correction can be performed even underconditions where optical conditions such as the lens being used areunknown, and also, crosstalk correction can be performed by digitalsignal processing.

With the invention according to Claims 4 through 6 of the presentapplication, in the case of using color coding in which white pixels andpixels of other colors such as RGB are adjacent, an evaluation value ofcrosstalk amount at a pixel to be corrected can be calculated based onthe proportion of the sum of the signal amount of adjacent RGB pixels asto the signal amount of white pixels, employing the fact that phenomenafrom the vertical direction and horizontal direction are dominant.

With the invention according to Claim 7 of the present application,crosstalk amount evaluation values can be calculated in real time bymaking N=4 for example, i.e., 4×4 pixels is the smallest increment. Inthis case, pixels of the same color within a processing increment havetwo or more outputs, so an average value of signal amount can be usedfor each color. Also, in the event that there is no need to calculatecorrection coefficients with fine granularity, or in cases where thereis no need to calculate correction coefficients in a single imaged imagesuch as with application to moving images, a relatively large blockaround N=100, i.e., 100×100 pixels may be used as the processingincrement.

With the invention according to Claim 8 of the present application,calculation of correction coefficients and crosstalk correction can beperformed using evaluation values calculated using previous frames, somoving image processing can be handled.

With the invention according to Claim 9 of the present application,correction coefficients can be calculated from evaluation values outputfrom the crosstalk amount calculating unit, based on a relationalexpression calculated beforehand between crosstalk amount evaluationvalues and correction coefficients.

With the invention according to Claim 10 of the present application, avalue obtained by multiplying the output signal of a pixel adjacent to apixel to be corrected is subtracted from the output signal of a pixel tobe corrected, whereby amount of crosstalk can be eliminated.

With the invention according to Claims 11 and 12 of the presentapplication, the degree of crosstalk over the entire imaging device facecan be known by evaluating each crosstalk amount using each array forcalculating evaluation values. Crosstalk can be suitably corrected bythen determining a correction coefficient in each region based on thecrosstalk amount evaluation value obtained from a nearby array forcalculating evaluation value.

Further objects, features, and advantages of the present invention willbecome apparent from detailed description made based on later-describedembodiments of the present invention and the attached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating the hardwareconfiguration of an imaging apparatus 100 serving as an embodiment ofthe present invention.

FIG. 2 is a diagram illustrating a functional configuration forperforming image signal processing for crosstalk correction.

FIG. 3A is a diagram illustrating an example of spectral properties ofan imaging device 12 for each color pixel (degree of crosstalk small)(spectral properties

FIG. 3B is a diagram illustrating an example of spectral properties ofthe imaging device 12 for each color pixel (degree of crosstalk medium)(spectral properties 2).

FIG. 3C is a diagram illustrating an example of spectral properties ofthe imaging device 12 for each color pixel (degree of crosstalk great)(spectral properties 3).

FIG. 4A is a diagram illustrating the way in which crosstalk occurs fromthe vertical direction and horizontal direction.

FIG. 4B is a diagram illustrating the way in which white signals mixinto adjacent RGB signals in the color pixel array shown in FIG. 12B.

FIG. 4C is a diagram illustrating the way in which RGB signals mix intoadjacent white signals in the color pixel array shown in FIG. 12B.

FIG. 5A is a diagram illustrating reflectance spectral properties of ablue patch (patch No. 13) in the Macbeth Color Checker chart.

FIG. 5B is a diagram illustrating reflectance spectral properties of agreen patch (patch No. 14) in the Macbeth Color Checker chart.

FIG. 5C is a diagram illustrating reflectance spectral properties of ared patch (patch No. 15) in the Macbeth Color Checker chart.

FIG. 5D is a diagram illustrating reflectance spectral properties of ayellow patch (patch No. 16) in the Macbeth Color Checker chart.

FIG. 5E is a diagram illustrating reflectance spectral properties of amagenta patch (patch No. 17) in the Macbeth Color Checker chart.

FIG. 5F is a diagram illustrating reflectance spectral properties of acyan patch (patch No. 18) in the Macbeth Color Checker chart.

FIG. 6A is a diagram illustrating the results of integrating thereflectance spectral properties of each Macbeth color path shown in FIG.5A through FIG. 5F with spectral properties corresponding to thecrosstalk amount for each color pixel of the imaging device 12 shown inFIG. 3A (spectral properties 1).

FIG. 6B is a diagram illustrating the results of integrating thereflectance spectral properties of each Macbeth color path shown in FIG.5A through FIG. 5F with spectral properties corresponding to thecrosstalk amount for each color pixel of the imaging device 12 shown inFIG. 3B (spectral properties 2).

FIG. 6C is a diagram illustrating the results of integrating thereflectance spectral properties of each Macbeth color path shown in FIG.5A through FIG. 5F with spectral properties corresponding to thecrosstalk amount for each color pixel of the imaging device 12 shown inFIG. 3C (spectral properties 3).

FIG. 7A is a diagram illustrating evaluation values obtained for eachMacbeth color patch for the primary six colors, with regard to eachspectral property according to the crosstalk amount (FIG. 3A throughFIG. 3C).

FIG. 7B is a diagram illustrating evaluation values obtained for allMacbeth color patches, with regard to each spectral property accordingto the crosstalk amount (FIG. 3A through FIG. 3C).

FIG. 8 is a diagram for describing a common crosstalk correctionprocessing method.

FIG. 9 is a diagram illustrating an example of a relational expressionbetween crosstalk amount evaluation values (K) and correctioncoefficients.

FIG. 10 is a diagram illustrating an example of a region (block) forcalculating correction coefficients with magnitude of a certain degree.

FIG. 11 is a diagram illustrating memory 4 for holding the evaluationvalue (K) calculated at a crosstalk correction calculating unit 1.

FIG. 12A is a diagram illustrating a Bayer array which is arepresentative filter array for a primary color system.

FIG. 12B is a diagram illustrating an example of a filter arrayincluding white pixels.

FIG. 13A is a diagram illustrating another example of a filter arrayincluding white pixels.

FIG. 13B is a diagram illustrating another example of a filter arrayincluding white pixels.

FIG. 13C is a diagram illustrating another example of a filter arrayusing a complementary color filter.

FIG. 14 is a diagram illustrating an example of a filter array whereinarrays including white pixels such as shown in FIG. 12B, are scatteredthroughout a Bayer array not including white pixels (see fog. 12A).

FIG. 15 is a diagram schematically illustrating the way in whichmultiple evaluation value calculating arrays such as shown in FIG. 12Bare disposed on an imaging device face based on the Bayer array shown inFIG. 12A.

FIG. 16 is a diagram schematically illustrating the way in whichmultiple evaluation value calculating arrays such as shown in FIG. 12Bare disposed on an imaging device face based on the Bayer array shown inFIG. 12A.

DESCRIPTION OF EMBODIMENTS

The following is a detailed description of embodiments of the presentinvention, with reference to the drawings.

FIG. 1 schematically illustrates the hardware configuration of animaging apparatus 10 serving as an embodiment of the present invention.Note that imaging apparatus as used here includes imaging devices,camera modules including an optical system for imaging image light on animaging face (light-receiving face) of the imaging device and a signalprocessing circuit for the imaging device, camera apparatuses such asdigital still cameras and video cameras in which the camera module isimplemented, and electronic equipment such as cellular phones.

In FIG. 1, image light from a subject (not shown) is imaged on theimaging face of an imaging device 12 by an optical system, an imaginglens 11 for example. For the imaging device 12, an imaging device isused which is formed by a great number pixels including photoelectricconverting devices being arrayed two-dimensionally in matrix fashion,and a color filter including color components of a primary color forcreating luminance components, and other color components, are disposedon the surface of the pixels. A color filter is a band-pass filter whichpasses light of predetermined wavelengths.

The imaging device having a color filter may be any of a charge-transferimaging device of which a CCD is representative, an X-Y address imagingdevice of which a MOS is representative, or the like.

Also, the color filter includes green (G) for example as a colorcomponents serving as a primary component for creating a luminance (Y)component, and red (R) and blue (B) for example as other colorcomponents, respectively, and performs color coding so as to reproducecolor of incident light at each pixel position. With the presentembodiment, color coding of an array including white pixels is performedfor the color filter, in order to realize high sensitivity and so forth.However, the array of pixels is not restricted to that shown in FIG.12B. Note that an arrangement may be made wherein as color componentsserving as a primary component for creating the Y component, white,cyan, yellow, or the like are used, and magenta, cyan, yellow, or thelike, are used for other color components.

With the imaging device 12, of the incident image light, only light ofeach color component passes through the color filters and is input toeach pixel. The light that has been input to each pixel is subjected tophotoelectric conversion by photoelectric converters such asphotodiodes. This is then read out from each pixel as analog imagesignals, converted into digital image signals at an A/D converter (ADC)13, and input to a camera signal processing circuit 14 which isequivalent to the image processing device according to the presentinvention.

The camera signal processing circuit 14 is configured of an opticalsystem correcting circuit 21, a WB (white balance) circuit 22, aninterpolation processing circuit 23, a gamma (γ) correction circuit 24,a Y (brightness) signal processing circuit 25, a C (chroma) signalprocessing circuit 26, a band limiting LPF (low-pass filter) 27, athinning out circuit 28, and so forth.

The optical system correcting circuit 21 performs correction of theimaging device 12 and optical system, such as digital clamping to matchthe black level with the digital image signals input to the camerasignal processing circuit 14, defect correction for correcting defectsof the imaging device 12, shading correction for correcting lightfalloff at edges for the imaging lens 11, and so forth.

As described above, the color filter used with the imaging deviceaccording to the present embodiment includes white pixels, so theproblem of crosstalk becomes pronounced, and accordingly there is theneed to perform correction thereof. While the point of performingcalculation and correction of crosstalk amount at the stage of digitalsignal processing is a main feature of the present invention, thefunction thereof cam be implemented within the optical system correctingcircuit 21. Details of calculation and correction of crosstalk amountwill be described later.

The WB circuit 22 subjects image signals which have passed through theoptical system correcting circuit 21 to processing for adjusting thewhite balance, such that RGB is the same as to a white subject. Theinterpolation processing circuit 23 creates pixels with differentspatial phases by interpolation, i.e., creates three planes from RGBsignals with spatially shifted phases (RGB signals at the same spatialposition).

The gamma (γ) correction circuit 24 subjects the RGB signals at the samespatial position to gamma correction, and then supplies to the Y-signalprocessing circuit 25 and C-signal processing circuit 26. Gammacorrection is processing for applying a predetermined gain to each ofthe R, G, and B color signals output from the WB circuit 22, such thatthe photoelectric conversion properties of the entire system, includingthe imaging device 12 and downstream image reproducing means and soforth, are 1, so as to correctly express the color tone of the subject.

The Y-signal processing circuit 25 creates brightness (Y) signals fromthe R, G, and B color signals, and the C-signal processing circuit 26creates Cr (R−Y) and Cb (B−Y) from the R, G, and B color signals.

The band limiting LPF 27 is a filter wherein the cutoff frequency f_(c)is ⅛ of the sampling frequency f_(s) for example, and drops the passingband for color difference signals Cr and Cb from (½) f_(s) to (⅛) f_(s).However, this is output for TV signal format, and in the event thatoutput is performed without band limitation, frequency signals of ⅛f_(s) or higher will be output as false color signals. The thinning outcircuit 28 performs thinning out of sampling of the color differencesignals Cr and Cb.

With the imaging apparatus 10 shown in FIG. 1, the color filter used forthe imaging device includes white pixels, so the problem of crosstalkbecomes pronounced. The present embodiment is configured thatcalculation and correction of the crosstalk amount is performed at thestage of digital signals correction. FIG. 2 illustrates the functionalconfiguration for performing image signal processing for crosstalkcorrection. The image signal processing is configured of a crosstalkamount calculating unit 1, a crosstalk correction coefficientcalculating unit 2, and a crosstalk correction unit 3, and isimplemented in the optical system correction circuit 21.

The crosstalk amount calculating unit 1 will be described first. Thecrosstalk amount calculating unit 1 performs quantification of thedegree of crosstalk as the crosstalk amount, based on imaged data outputfrom the imaging device 12.

FIG. 3 illustrates examples of spectral properties for each color pixelof the imaging device 12. In the example shown in the drawing, thedegree of crosstalk increases in the order of FIG. 3A, FIG. 3B, and FIG.3C. Blue (B) is a filter which passes around 450 nanometers, Green (G)is a filter which passes around 550 nanometers, and red (R) is a filterwhich passes around 650 nanometers. Also, white (W) pixels are the sameas with a monochrome imaging device with no color filter. When thecrosstalk amount increases, the output at frequency regions where thereshould be no sensitivity increases. For example, in FIG. 3C, the outputand the band of 550 to 650 nanometers has increased at the waveform forblue (B_3) pixels, due to crosstalk.

Now, with a color filter array such as shown in FIG. 12B, the whitepixels and each of the RGB pixels are next to each other. Generally,with crosstalk, phenomena from the vertical direction and horizontaldirection are dominant as shown in FIG. 4B. Crosstalk can be generallydivided into two types, i.e., one where white signals are mixed intoadjacent RGB signals, as shown in FIG. 4B, and one where RGB signals aremixed into adjacent white signals, as shown in FIG. 4C.

Using the nature shown in FIG. 4 enables the crosstalk amount to beknown in a relative manner. The method thereof is to calculate theproportion between the sum of the signal amount of each of the signals,and the signal amount of the white signals (described later).

Now, in the field of color imaging, including digital cameras foremost,generally a “Macbeth Color Checker (Macbeth Color chart)” is used forevaluating color reproducibility. For example, “Color Imaging”, editedby the Color Science Association of Japan (pp 29-33) describes thatspectral sensitivity, tone reproduction, and the three primary colorsare factors governing color reproducibility, and that a method isgenerally used in which these factors are not separately evaluated incolor reproducibility evaluation but rather the color reproducibilityfinally obtained is evaluated, and that as for the evaluation method, astandard color chart is input as an image and the output reproducedcolors are compared with the colors of the original color chart byspectral reflectivity (transmissivity), and that the Macbeth Color chartis widely used as the color chart. A Macbeth Color chart is made up of24 colors including 6 shades of gray. The surface of each color chart ismatte, and is of a size of 45 mm×45 mm. This literature lists thereflective spectral properties (spectral reflectivity) of the MacbethColor chart as appendix Tables A.1 and A.2. Description will be madebelow using this spectral data.

FIGS. 5A through 5F illustrate the reflective spectral properties ofeach patch of blue (patch No. 13), green (patch No. 14), red (patch No.15), yellow (patch No. 16), magenta (patch No. 17), and cyan (patch No.18) in the Macbeth Color Checker chart. Note that the reason that onlythe above six colors of the 24 colors in the Macbeth Color chart areused is due to the fact that these six colors are the primary colorcomponents used in many color imaging systems.

Multiplying the reflectance spectral properties of these Macbeth Colorcharts (FIG. 5A through FIG. 5F) by the spectral properties of the colorfilters of the imaging device 12 shown in FIGS. 3A through 3C at eachwavelength component and obtaining the sum, i.e., integration thereof,represents the output of each floor form the imaging device 12.

FIG. 6A through FIG. 6C illustrate the results of integrating thereflectance spectral properties of each Macbeth color patch shown inFIG. 5A through FIG. 5F with the spectral properties of each color pixelof the imaging device shown in FIG. 3A through FIG. 3C, respectively.Put simply, FIG. 6A through FIG. 6C are equivalent to outputcorresponding to the crosstalk amount of each color pixel of the imagingdevice 12.

An evaluation value (K) for evaluation the crosstalk amount can becalculated using the following Expression (1) for example, based on theoutput (see FIG. 6A through FIG. 6C) for each of the color pixels (R, G,B, W) of the imaging device 12 obtained from the spectral properties(see FIG. 3A through FIG. 3C) according to the crosstalk amount of eachcolor pixel of the imaging device 12.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 1} \right\rbrack & \; \\{K = \frac{{\alpha \; R} + {\beta \; G} + {\gamma \; B}}{ɛ\; W}} & (1)\end{matrix}$

In the above Expression (1), R, G, b, and W are output values of each ofthe color pixels (see FIG. 6A through 6C), α, β, γ, and ε are arbitrarycoefficients, and the evaluation value (K) is equivalent to the resultof calculating the proportion of the sum of output of each of the RGBcolor pixels as to the output for the white pixels. This expression isbased on the fact that with the color coding shown in FIG. 12B forexample, crosstalk can be generally divided into two types; one wherewhite signals are mixed into adjacent RGB signals, as shown in FIG. 4B,and one where RGB signals are mixed into adjacent white signals, asshown in FIG. 4C (described above).

Calculation of the evaluation value (K) shown in the above Expression(1) is performed for output according to crosstalk amount of each colorpixel of the imaging device 12 (see FIG. 6A through 6C), for each patchof the Macbeth Color chart, whereby an evaluation value can be obtainedfor each Macbeth color patch, with regard to each spectral propertycorresponding to crosstalk amount (FIG. 3A through FIG. 3C) can beobtained.

FIG. 7A illustrates evaluation values obtained for each Macbeth colorpatch, with regard to each spectral property corresponding to crosstalkamount (FIG. 3A through FIG. 3C). Also, the average and standarddeviation for the evaluation values K_1, K_2, and K_3 obtained for eachof the spectral properties 1 through 3 over all six color patches havebeen compiled in the following table.

TABLE 1 Average Standard deviation K_1 1.06 0.016 K_2 1.15 0.012 K_31.35 0.027

From the above table, it can be seen that the evaluation value (K) isgenerally constant, regardless of the reflectance properties of thesubject (each color). This means that the evaluation value (K)calculated from the above Expression (1) is capable of being used inevaluating crosstalk amount.

Note that the coefficients α, β, γ, and ε are optimized such that theevaluation value (K) is constant in ideal spectral properties where thecrosstalk amount is small, as shown in FIG. 3A, for example. In reality,an approximation method such as least square or the like is used. Theobtained value is used as to other spectral properties such as in FIG.3B and FIG. 3C, as well. In FIG. 7A, the evaluation value (K) has beencalculated using the coefficient values shown in the followingExpression (2).

[Mathematical Expression 2]

α=β=γ=ε=1   (2)

The reason that only the above six colors of the 24 colors in theMacbeth Color chart are used in the above description is due to the factthat these six colors are the primary color components used in manycolor imaging systems (described above). The present inventors performedcalculation of evaluation values for each of the spectral properties,using the Macbeth Color chart for all 24 colors, for the sake ofassurance. FIG. 7B illustrates the results. Also, the average andstandard deviation for the evaluation values K_1, K_2, and K_3 obtainedfor each of the spectral properties 1 through 3 over all 24 colorpatches have been compiled in the following table. Since the evaluationvalue (K) is generally constant regardless of the reflectance propertiesof the subject (each color), it can be reconfirmed that the evaluationvalue (K) calculated from the above Expression (1) is capable of beingused in evaluating crosstalk amount.

TABLE 2 Average Standard deviation K_1 1.07 0.010 K_2 1.16 0.011 K_31.35 0.025

With the above description, it can be understood that relative change incrosstalk amount can be detected by calculating the evaluation value (K)using output signals from the imaging device 12 using color coding inwhich white pixels are added to RGB pixels. That is to say, the degreeof crosstalk amount can be detected from the output signals of theimaging device 12 alone, with no need to measure the crosstalk amountwithin the chip beforehand as has been conventional done (e.g., see PTL2). Accordingly, the degree of crosstalk can be quantized at the stateof digital signal processing, even in a situation wherein opticalconditions, such as the lens to be used, are unknown.

With the crosstalk amount calculating unit 1, output signals of pixelsof all colors including the white pixels are necessary, as can beunderstood from the above Expression (1). Accordingly, in the case ofcalculating the evaluation value (K) in real time as to the imagingdevice 12 having a filter array such as shown in FIG. 12B, the value ofaround 4×4 pixels is preferably handled as the minimum increment. Pixelsof the same color within a processing increment have two or moreoutputs, so the average value of signal amount is preferably used tocalculate the above Expression (1).

Next, the crosstalk correction coefficient calculating unit 2 will bedescribed. At the crosstalk correction coefficient calculating unit 2, acrosstalk correction coefficient corresponding to the current crosstalkamount is calculated from the crosstalk amount output from the crosstalkamount calculating unit 1 and a relational expression between thecrosstalk correction coefficient and crosstalk amount obtainedbeforehand.

First, a general crosstalk correction processing method will bedescribed with reference to FIG. 8. As shown in FIG. 4A, with crosstalk,phenomena from the vertical direction and horizontal direction aredominant. Accordingly, in other words, several tenths of the signals ofadjacent pixels vertically and horizontally can be subtracted from thesignal of the pixel to be corrected as crosstalk amount. The outputsignals of a pixel to be corrected can be corrected by the followingExpression (3).

[Mathematical Expression 3]

S _(—) crct(i,j)=S(i,j)−a·S(i,j−1)−b·S(i−1,j)−c·S(i+1,j)−d·S(i,j+1)  (3)

In the above Expression (3), S_crct represents the signal aftercorrection, S represents the signal before correction, and in side theparentheses are the coordinate positions, respectively. (i, j) is onecoordinates of the pixel to be corrected. Also, a, b, c, and d arecorrection coefficients as to adjacent pixels above, left, right, andbelow. These a, b, c, and d are also values indicating the proportion ofthe adjacent pixel signals being crosstalk amount.

In the event that the crosstalk amount is constant regardless ofsheeting conditions or pixel position within the chip, the correctioncoefficients a, b, c, and d may also be constant. However, in reality,the crosstalk amount changes depending on the color temperature of thelight source and optical conditions, and pixel position within the chip.Generally, as the crosstalk amount increases, the correctioncoefficients also become greater.

Accordingly, with the present embodiment, shooting is performedbeforehand changing the optical conditions, illumination colortemperature conditions, and so froth, correction coefficients arecalculated corresponding to the output of the crosstalk amountcalculating unit 1, i.e., to the evaluation value (K), and a relationalexpression such as shown in FIG. 9 is created. With the crosstalkcorrection coefficient calculating unit 2, upon the evaluation value (K)being output from the crosstalk amount calculating unit 1, such arelational expression is referenced to obtain correction coefficientscorresponding to the crosstalk amount in the area where shooting isactually being performed.

Finally, description will be made regarding the crosstalk correctionunit 3. As described above, with crosstalk, phenomena from the verticaldirection and horizontal direction are dominant (see FIG. 4A).Accordingly, with the crosstalk correction unit 3, the signal of thepixel to be corrected is corrected by subtracting several tenths of theeach of the signals of pixels adjacent vertically and horizontally, fromthe signal of the pixel to be corrected as crosstalk amount, followingthe correction expression shown in Expression (3) above for example,using the correction coefficients a, b, c, and d of the adjacent pixelscalculated by the crosstalk correction coefficient calculating unit 2.

So far, a method has been described for calculating crosstalk correctioncoefficients for pixels to be corrected, with a size of around 4×4pixels as the minimum increment. However, in reality there are caseswherein there is no need to calculate correction coefficients with sucha fine granularity. Accordingly, a method for calculating correctioncoefficients with an image beforehand, or with a certain size, to handleprocessing of moving images, will be described below.

FIG. 10 illustrates an example of a region (block) for calculatingcorrection coefficients with a certain size. In the example in thedrawing, we will say that each block is made up of 100×100 pixels, andone imaged image is made up of 6×8 blocks.

At the crosstalk amount calculating unit 1, upon calculating the averageof the pixel values of each color of and white as processing for eachblock, the evaluation value (K) is calculated following the aboveExpression (1). Then at the downstream crosstalk correction coefficientcalculating unit 2 and crosstalk correction unit 3, calculation ofcorrection coefficients and pixel value correction processing are eachperformed.

Now, as shown in FIG. 11, we will say that memory 4 is provided to holdthe evaluation value (K) calculated at the crosstalk amount calculatingunit 1. The evaluation value (K) calculated at the crosstalk amountcalculating unit 1 is then saved in the memory 4, the evaluation value(K) is updated at a certain number of fixed intervals, and correctionprocessing is performed on the imaged data. With 100×100 pixel blocks asthe minimum increment, pixels of the same color within a processingincrement have two or more outputs, so the average value of signalamount is preferably used to calculate the above Expression (4). Thenumber of pixels for calculating the evaluation value (K) is great, soeven if there is much noise included in the data, an accurate evaluationvalue (K) can be obtained by averaging.

$\begin{matrix}{\mspace{76mu} \left\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 4} \right\rbrack} & \; \\{K = \frac{{{average}\mspace{14mu} {of}\mspace{14mu} \alpha \; R} + {{average}\mspace{14mu} {of}\mspace{14mu} \beta \; G} + {{average}\mspace{14mu} {of}\mspace{14mu} \gamma \; B}}{{average}\mspace{14mu} {of}{\mspace{11mu} \;}W\; ɛ}} & (4)\end{matrix}$

Also, in the event that the difference in correction is conspicuous atboundary portions of the blocks in the image following having performedcrosstalk correction, this portion can be made inconspicuous byaveraging the correction coefficients between the adjacent blocks.

While description has been made so far regarding an embodiment of thepresent invention using the example shown in FIG. 12 as a filter arrayincluding white pixels, but the essence of the present invention is notrestricted by color coding. For example, the evaluation value forknowing crosstalk amount can be calculated from shooting data alone forarrays with different RGB arrays such as shown in FIG. 13A and FIG. 13B,or arrays using complementary color filters instead of primary colorfilters as shown in FIG. 13C, for example, in the same way as describedabove, and crosstalk correction processing can be performed usingcorrection coefficients adapted to his evaluation value.

Also, the present invention performs crosstalk correction of each pixelbased on the evaluation results of crosstalk amount of white signals asto adjacent RGB signals (see FIG. 4B) and crosstalk amount of RGBsignals as to adjacent white signals (see FIG. 4C), and in other words,white pixels are necessary for evaluating the crosstalk amount.

However, there is no need for the white pixels to be uniformly arrayedover the entire imaging device face, and calculation of crosstalkevaluation values can be performed simply by disposing white pixels injust a partial manner. FIG. 14 illustrates an example of a filter arrayin which arrays including white pixels such as shown in FIG. 12B arescattered throughout a Bayer array not including white pixels (see FIG.12A). In such a case, crosstalk amount can be obtained from the arrayshown in FIG. 12B, and the crosstalk within the Bayer array can becorrected based on the crosstalk correction coefficients calculatedusing this crosstalk amount.

FIG. 15 schematically illustrates the way in which multiple evaluationcalculating arrays such as shown in FIG. 12B are disposed on the imagingdevice face based on a Bayer array (see FIG. 12A). Evaluating eachcrosstalk amount using each evaluation calculating array allows thedegree of crosstalk to be known over the entire face of the imagingdevice. Crosstalk then can be adaptively corrected in each region bydeciding crosstalk correction coefficients based on the evaluation valueof the crosstalk amount obtained from a nearby evaluation calculatingarrays.

Now, the relation between crosstalk amount evaluation value andcorrection coefficients is obtained beforehand, as shown in FIG. 9. Therelation between the evaluation value and correction coefficients may beobtained for each evaluation calculating arrays.

The crosstalk correction in each region can be performed using the aboveExpression (3). Alternatively, crosstalk correction may be performedfollowing the matrix operation shown in the following Expression (5),after performing interpolation processing at the interpolationprocessing circuit 23 (see FIG. 1).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 5} \right\rbrack & \; \\{\begin{pmatrix}R_{i}^{\prime} \\G_{i}^{\prime} \\B_{i}^{\prime}\end{pmatrix} = {\begin{pmatrix}R_{11} & G_{12} & B_{13} \\R_{21} & G_{22} & B_{23} \\R_{31} & G_{32} & B_{33}\end{pmatrix}\begin{pmatrix}R_{i} \\G_{i} \\B_{i}\end{pmatrix}}} & (5)\end{matrix}$

-   R′, G′, B′: Signals following correction-   R, G, B: Signals before correction-   R₁₁, G₁₂, B₁₃, R₂₁, G₂₂, B₂₃, R₃₁, G₃₂, B₃₃: Correction coefficients-   i: position of pixel in imaging device array

In the event that the pixel array of the imaging device is partiallydifferent as shown in FIG. 15, the interpolation circuit 23 needs to bechanged. The interpolation method can be switched at predetermined pixelpositions for this changing. FIG. 16 illustrates a configuration exampleof an imaging device for switching the interpolation method according tothe pixel position. Note however, that only relevant parts are extractedand shown in this drawing. Interpolation processing can be performed atan interpolation circuit 23A at pixel position following the Bayerarray, and switched to interpolation processing at the interpolationcircuit 23 for pixel positions for the evaluation value calculationarrays (see FIG. 12B).

INDUSTRIAL APPLICABILITY

While the present invention has been described in detail with referenceto a particular embodiment, it is self-evident that one of ordinaryskill in the art can make modifications and substitutions to theembodiment without departing from the essence of the present invention.The present invention can be applied to, for example, a camera apparatussuch as a digital still camera or video camera, various types ofelectronic equipment in which a camera module is implemented, such ascellular telephones, and so forth.

While an embodiment of the present invention has been described in thePresent Description using the example shown in FIG. 12 as a filter arrayincluding white pixels, the essence of the present invention is notrestricted to this. For example, the evaluation value for knowingcrosstalk amount can be calculated from shooting data alone for arrayswith different RGB arrays such as shown in FIG. 13A and FIG. 13B, orarrays using complementary color filters instead of primary colorfilters as shown in FIG. 13C, for example, in the same way as describedabove, and crosstalk correction processing can be performed usingcorrection coefficients adapted to this evaluation value.

In short, the present invention has been disclosed exemplarily, and thecontents of description within the Present Description should not beinterpreted restrictively. The Claims should be taken into considerationto determined the essence of the present invention.

REFERENCE SIGNS LIST

1 crosstalk amount calculating unit

2 crosstalk correction coefficient calculating unit

3 crosstalk correction unit

10 imaging apparatus

11 imaging lens

12 imaging device

13 A/D converter (ADC)

14 camera signal processing circuit

21 optical system correcting unit

22 WB (white balance) circuit

23 interpolation processing unit

24 gamma correction circuit

25 Y-signal processing circuit

26 C-signal processing circuit

27 band limiting LPF (low-pass filter)

28 thinning out circuit

1. An image processing device comprising: a crosstalk amount calculatingunit for calculating an evaluation value of crosstalk amount included inan output signal from a pixel to be corrected in an imaging device; acrosstalk correction coefficient calculating unit for calculating acrosstalk correction coefficient based on the evaluation value outputfrom said crosstalk amount calculating unit; and a crosstalk correctingunit for eliminating crosstalk amount included in an output signal ofsaid pixel to be corrected, using said crosstalk correction coefficient.2. The image processing device according to claim 1, wherein saidcrosstalk amount calculating unit calculates the evaluation value ofcrosstalk amount included in an output signal of said pixel to becorrected, based on output signals from the imaging device.
 3. The imageprocessing device according to claim 1, wherein said crosstalk amountcalculating unit calculates the evaluation value of crosstalk amountincluded in an output signal of said pixel to be corrected, based on therelation of output signals between adjacent pixels.
 4. The imageprocessing device according to claim 1, wherein said imaging device usescolor coding including white pixels.
 5. The image processing deviceaccording to claim 4, wherein said crosstalk amount calculating unitcalculates an evaluation value for crosstalk amount included in anoutput signal of a pixel to be corrected, based on the proportion of thesum of the signal amount of the pixels other than white, as to thesignal amount of white pixels.
 6. The image processing device accordingto claim 4, wherein said crosstalk amount calculating unit calculates anevaluation value for the relative amount of crosstalk included in anoutput signal of a pixel to be corrected, based on the proportion of thesum of values obtained by multiplying the signal amounts of each of RGBpixels by respective predetermined coefficients (α, β, γ), as to a valueobtained by multiplying the signal amount of white pixels by apredetermined coefficient (ε).
 7. The image processing device accordingto claim 4, wherein said crosstalk amount calculating unit calculates anevaluation value of crosstalk amount, with N×N pixels as an increment ofprocessing (where N is a positive integer).
 8. The image processingdevice according to claim 7, further comprising memory for storingevaluation values which said crosstalk amount calculating unit hascalculated in increments of processing; wherein said crosstalkcorrection coefficient calculating unit and said crosstalk correctionunit respectively perform calculation of correction coefficients andcorrection of crosstalk, using the evaluation values calculated usingprevious frames saved in said memory.
 9. The image processing deviceaccording to claim 1, wherein said crosstalk correction coefficientcalculating unit calculates beforehand a relational expression betweenthe evaluation value of crosstalk amount calculated by said crosstalkamount calculating unit, and correction coefficients, and at the time ofan evaluation value output from said crosstalk amount calculating unitbeing output, references said relational expression and calculates acorrection coefficient corresponding to said evaluation value.
 10. Theimage processing device according to claim 1, wherein said crosstalkcorrection unit subtracts, from an output signal of a pixel to becorrected, a value obtained by multiplying the output signal of a pixeladjacent to said pixel to be corrected by said correction coefficient,thereby eliminating amount of crosstalk.
 11. The image processing deviceaccording to claim 1, wherein said imaging device includes a pluralityof arrays for calculating evaluation values including white pixels, inan array not including white pixels.
 12. The image processing deviceaccording to claim 11, wherein said crosstalk amount calculating unituses each of said arrays for calculating evaluation values to calculateevaluation values of crosstalk amount occurring at relevant positions;and wherein said crosstalk correction coefficient calculating unitcalculates a crosstalk correction coefficient based on the evaluationvalue output from said crosstalk amount calculating unit, for eachposition where an array for calculating evaluation values is disposed;and wherein said crosstalk correcting unit performs correction ofcrosstalk using a relevant coefficient, within an array for calculatingevaluation values, and performs correction of crosstalk using acrosstalk correction coefficient determined based on an evaluation valueof crosstalk amount obtained from a nearby array for calculatingevaluation values, in a region outside of an array for calculatingevaluation values.
 13. An image processing method comprising: acrosstalk amount calculating step for calculating an evaluation value ofcrosstalk amount included in an output signal from a pixel to becorrected in an imaging device; a crosstalk correction coefficientcalculating step for calculating a crosstalk correction coefficientbased on the evaluation value output in said crosstalk amountcalculating step; and a crosstalk correcting step for eliminatingcrosstalk amount included in an output signal of said pixel to becorrected, using said crosstalk correction coefficient.
 14. An imagingapparatus comprising: an imaging device including a color coding colorfilter; and a signal processing unit for processing output signals ofsaid imaging device; wherein said signal processing unit includes acrosstalk amount calculating unit for calculating an evaluation value ofcrosstalk amount included in an output signal from a pixel to becorrected in said imaging device, a crosstalk correction coefficientcalculating unit for calculating a crosstalk correction coefficientbased on the evaluation value output from said crosstalk amountcalculating unit, and a crosstalk correcting unit for eliminatingcrosstalk amount included in an output signal of said pixel to becorrected, using said crosstalk correction coefficient.
 15. A computerprogram described in a computer-readable format so as to executeprocessing of output signals from an imaging device having a colorcoding color filter, said computer program causing said computer tofunction as: a crosstalk amount calculating unit for calculating anevaluation value of crosstalk amount included in an output signal from apixel to be corrected in said imaging device, a crosstalk correctioncoefficient calculating unit for calculating a crosstalk correctioncoefficient based on the evaluation value output from said crosstalkamount calculating unit, and a crosstalk correcting unit for eliminatingcrosstalk amount included in an output signal of said pixel to becorrected, using said crosstalk correction coefficient.